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Abstract 

The purpose of this article is to show how the isotropy subgroup of leaf permutations on bi- 
nary trees can be used to systematically identify tree-informative invariants relevant to models of 
phylogenetic evolution. In the quartet case, we give an explicit construction of the full set of rep- 
resentations and describe their properties. We apply these results directly to Markov invariants, 
thereby extending previous theoretical results by systematically identifying linear combinations 
that vanish for a given quartet. We also note that the theory is fully generalizable to arbitrary 
trees and is equally applicable to the related case of phylogenetic invariants. All results follow 
from elementary consideration of the representation theory of finite groups. 
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1 Preliminaries 



Phylogcnctic methods seek to reconstruct the evolutionary history of organisms from present-day 
data such as DNA and are of fundamental importance in the biological sciences ([Felsensteinl . 
I2OO4I ). Approaches to this importa nt problem draw upon sophisticated mathematical, statistical 
and computational techniques (see iGascuell (|2005r ) for an overview) . From a purely theoretical 
point of view, this represents a wonderful confluence of hitherto disparate areas of mathematics. 
In particular, models of phylogenetic evolution require a marriage between gra ph theory, com- 
binato rics and stochastic processes (a comprehensive treatment can be found in ISemple fc Steel 
There is also a rich algebraic structure underlying phylogenetic models - particularly 
when the complications of working with binary trees is taken i nto account. For instan ce, spectral 
analysis of the Kimura 3ST model using Hada mard conjugation iHendv fc Penny , 19891 ) and group 
based approaches to phylogenetic invariants ( Evans fc Speed . 1993 ) provide novel applications of 
algebra to phylogenetics. This article serves as a direct s equel to the al g ebraic approach apply- 
ing group representation theory to phylogenetics given in ISumner et al] ( 2008 ). where "Markov 
invariants" were defined and explored. 

Standard stochastic models of phylogenetic evolution are high-dimensional, with the number 
of free parameters being proportional to the number of leaves on the evolutionary tree. Given that 
DNA sequences are of finite extent, it follows that phylogenetic data sets are often quite sparse 
and significant model- f i tting problems arise with respect to the issue of bias/ variance trade-off 



( Burnham fc Anderson . 2002f) . In this light, Markov invariants provide one- dimensional "repre- 



sentations" of these stochastic models that retain some of the complex structure of these models, 
while greatly reducing the number of free parameters present. Significantly, Markov invariants are 
defined to respect the infinitesimal unfolding of a continuous-time Markov chain. This property 
is not stipulated in t he definition of phylogenetic invariants and there is some evidence (given in 
Sumner et that this addi t ional structure can assist in the search for "powerful" sets 



of phylogenetic invariants ( Erikssoiil 2008 ). In particular, it should be noted that the popular 
Log-Det pairwise distance Isteell . I1994I ) has as its foundation the simplest example of a Markov 
invariant. 

We say that a Markov invariant is "tree-informa t ive" i f it satisfies the conditions of a phyloge- 
netic invariant (jCavender fc Felsensteinl . ll987l : lLakd . [l987h for particular trees. Here we show how 
to systematically find linear combinations of Markov invariants that are tree-informative. An ex- 
plicit construction is given in the case of quartet trees by studying the irreducible representations 
of the isotropy subgroup of leaf permutations on quartets. 

Presently we review some basic concepts and terminology from lSumner et all ( 2008h . 

Given a group Q, recall that a group representation is a homomorphism p : Q ^ GL{V), where 
GL{V) is the set of invertible linear operators on a vector space V. This provides an action of G 
on V and in this case V is referred to as a Q-module (or, a module of Q, or, when the group is 
understood, simply, a module). [/ C y is said to form an invariant subspace if it is closed under 
the action of Q, i.e. p{G) ■ U C U. 

In this article, a tree T is a connected acyclic graph with vertices of valence 3 or 1 only. The 
vertices of valence 1 are referred to as leaves and are denoted by L with m:=\L\. All results given 
will be relevant to the general Markov model ( Allman fc Rhod^ . 2003) of sequence evolution on a 
tree (including the IID assumptions) , with the addit i onal c onstraint that all transition matrices are 



chosen from the Markov semigroup ( Sumner et al. . 2008f l. Restricting to the Markov semigroup 



ensures that the process arises as a continuous-time Markov chain, and allows us to refer to notions 
of continuity and the infinitesimal. We denote elements of the Markov semigroup as Ma and 
employ right multiplication so that the matrix element m-j""* ■— [Ma\j^ represents the probability 
of a transition i j. 

In particular, consider random variables defined at the leaves of a tree Xi , X2 , . . . , . We 
suppose these random variables take on one of k discrete values with an associated probability 
distribution 



Piii2-..im ^ [^1 — *l7 -'^2 — 42, • • • X„i—im] 
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Given the fc-dimensional vector space V = C'^ with basis vectors {ei}i<i<k, the phylogenetic tensor 
P G y®"* is defined as 

P ■= ^ Kii2...i™eii (g) . . . ® ei^. 

l<ii .12;. ..^ijn'^k 

If this distribution is generated under a Markov assumption (as is standard for phylogenetic 
models), the "local" (no branching events) change of this tensor is described by 

F' = .g • P Ml ® Ma . . . ® M™ • P, (1) 

where each Mi is an element of the Markov semigroup. Markov invariants (of weight w) are defined 
as functions that take a simple form under this local change: 

/(P') fig ■ P) = det(g)-7(P). 

As each term in detfq) = det(Mi ) . . . det (Mm,) can be related to expected number of state changes 
under the model (jSemple fc Steej [20031 chap. 8), we see that a Markov invariant reduces the high- 
dimensionality of ([1]) to a single parameter that is related to the total number of state changes 
expected from this process. However, as it stands, this definition of Markov invariants says nothing 
about any underlying tree structure. It is rectifying this situation that is the main purpose of this 
article. 

The definition can be viewed as a group action on the Markov invariants themselves by setting 
{g^^ of) (P) ■— f{gP). Thus a Markov invariant transforms under the Markov process as a 
one-dimensional module of the Markov semigroup: 

g-^of^Aet{grf. 

It should be noted that existence of g~^ is guaranteed as all elements of the Markov semigroup 
are invertible as linear operators (we return to this point in the next section). 

By applying Schur-Weyl duality between t he symmetric and the general linear groups, existence 
conditions for such invariants were given in Sumner et al. ( 20081) using inner multiplications of 



Schur functions. In particular, in the case of DNA and quartet trees, fc = 4 and m = 4, it was 
shown that there exist four linearly independent Markov invariants of degree d=5. 

In this article we extend these results by including the "global" aspect of the tree and branching 
process thereof. Previously this has been achieved by laboriously checking (with a computer) for 
linear relations between Markov invariants when evaluated on canonical forms of phylogenetic 
tensors arising from different trees. This procedure identified linear combinations of Markov 
invariants that vanish for certain trees, hence producing tree-informative invariants that satisfy the 
usual definition of phylogenetic invariants along with respecting the local transformation properties 
discussed above. Here we will achieve the same result by studying the transformation properties 
of Markov invariants under leaf permutations. 



Rather than deal with the automorphism group of a tree (jGodsil fc Rovld . 120011 ). we consider 
the isotropy subgroup Gr of leaf permutations &m — Sym(L). Formally this corresponds to the 
automorphism group restricted to the leaf vertices: 



gT= Ant {T)\ 



L ■ 



Although it is clear that as abstract groups we have Qt = Aut(T) (under the action of an element 
of Aut(T) the images of the leaves uniquely determines the image of each internal vertex), it is 
crucial to our discussion to make this distinction so that Gt can be viewed as a subgroup of the 
symmetric group &m- This allows us to define an action of Gt on the space of phylogenetic tensors 
and respects the underlying biology, as it is the labelling of vertices at the leaves that is of primary 
importance. 

In what follows we will deal with the simplest non-trivial case: quartets. We will derive the 
multiplication table for the isotropy group of a quartet, compute its conjugacy classes, irreducible 
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Figure 1: Unrooted, leaf-labelled quartet trees 



representations, character table, and group branching rule upon restriction from 64. In doing so 
we completely characterize the quartet case and give a clear path to the general theory for larger 
trees. All results are applied to Markov invariants, but it should be noted that the technique 
presented is directly relevant to other structures that arise in phylogenetics including, of course, 
phylogenetic invariants. 

2 Isotropy subgroups of quartets 

Consider the three possible unrooted leaf-labelled quartet trees given in Figure[TJ We can represent 
each of these quartets as a word from the alphabet {"1","2","3","4","|"} in several ways: 





12|34 = 


21|34^ 


34|12 


T2 


:= 13|24 = 


31|24 = 


24|13 


% 


:= 14123 = 


41123 = 


23114 



An action of the symmetric group S4 on these words is defined by permuting the leaf labels: 

ij\kl I— > cr • ij\kl — a{i)a{j)\a{k)a{l), Vcr G 64. 

For example, using the cycle notation for the symmetric group we have 

(12) • Ti = (12) • 12|34 = 21|34 = 12|34 = Ti, 
(123) • Ti = (123) • 12|34 = 23|14 14|23 = T3, 

and 

(13)(24) • Ti = (13)(24) • 12|34 34|12 ^ 12|34 ^ Ti. 

This group action actually defines a homomorphism 64 — > ©3, as 64 acts by permuting the three 
quartets. However, this homomorphism will not be of primary interest to us. 

Given a group Q acting on a set X , the isotropy subgroup of the element x E X is defined 
as the set of group elements that leave x fixed: 

Gx -"^ {g ^ G\g ■ X = x} . 

It is easy to show that Gx does indeed form a subgroup. (The reader should note that some authors 
refer to an isotropy subgroup as a "stabilizer" subgroup.) 

We are interested in the isotropy subgroup of each of the quartet trees: 

012134 := {<J e 64 k- 12|34^ 12|34}, 

with Gi3\24 s^nd 5i4|23 defined similarly. By exhaustive search through the elements of 64, we find 
that 

012134 = {e, (12), (34), (12)(34), (13)(24), (14)(23), (1324), (1423)}, 

where e denotes the identity element. This subgroup can be generated from the elements (1324) 
and (13)(24) so that any element can be expressed as a product of these two. If we set a=(1324) 
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and b = (13) (24) we find that ^ ~ e and b~^ab = a^^ . In this way we see that Qi2\34 is 
isomorphic to the dihedral group D^; the symmetry group of a square. 

Recall that, for finite trees, a "rotation" is defined as an element of Aut(T) (excluding the 
i dentity) that fixes a t least one vertex of T, whereas a "reflection" flips at least one internal edge 
( Gawron et al. 1. 1 19991) . Thus, referring to Figure[l]we see that (12), (34) and (12)(34) are rotations. 



while (13)(24), (14)(23), (1324) and (1423) are reflections. 

In this article we consider phylogenetic tensors that are constructed using transition matrices 
chosen from the Markov semigroup. Recall that every element M of the Markov semigroup satisfies 
< det(M) < 1 , with det(M) — 1 occurring only in the trivial case where M is the identity operator 
( Sumner et MI, l2008h . Thus if we assume that all transition matrices are non-trivial, thereby 



ensuring non-zero branch lengths and binary evolutionary trees, we can apply identifiability of tree 
topology (Chang, 1996) and conclude that the phylogenetic tensors on quartets can be partitioned 
into disjoint subsets, with each subset corresponding to a quartet. Thus, if we denote the set of 
phylogenetic tensors as V"-^' C 1/®^, where 7^ is a quartet and = C'^, we have: 

It should be noted that these are subsets and clearly not subsjoaces of the vector space V ^^. In 



It should be noted that these are subsets and clearly not subsjoaces of the vector space V ^^. In 
fact, the recent non- identifiability result for phylogenetic mixtures of Matsen fc Steel ( 200?! ) imply 



that each V'^^ is not even closed under real, convex linear combinations. However, this will not 
affect any of the results discussed in the present work: we will simply have to replace the phrase 
"invariant subspace" with "invariants subset", where relevant. 
There is an action of 64 on 1/®^ defined as 

Informally, this is equivalent to writing 

"■■'041424344 = V'lff ( i) Ijj (2) iff (3) iff (4) ' (2) 

where, for ease of reading, we have set a = a^^ . Clearly this induces an action of Qyiyi^ on the 
set of phylogenetic tensors. 

Lemma 2.1. V'^^ forms an invariant subset under the action of 012134: ■ Further, 

if sgn{a) — 1, and 

if sgn{a) = -1, for all a e ^i2|34- 

Proof. This result follows easily by noting that Gi2\34 ■ T^i = Ti hy definition, and checking that 
(T • 72 = 72 if sgn((T) = 1 and cr • 72 = 7i3 if sgn((T) = — 1. However, we confirm the proof explicitly 
to illustrate the way the symmetric group acts on phylogenetic tensors. 

The components of any phylogenetic tensor P £ V'^'^ can be expressed as 

E(l) (2) (3) (4) (0) 
'^4i>42>i3>i4i'^}/'"i' 

l<i,j<k 

where, for each a, nrS°^ are the matrix elements of an element Ma of the Markov semigroup. 
We have (arbitrarily) chosen to root the quartet at the parent vertex of leaf 1 and 2 with root 
distribution tt (see Figured]). 
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Figure 2: Quartet tensor 



The "trimmed" tensor P ( Sumner et ai . 2008 ) is generated from P by trimming off the pendant 



edges of the tree or, more precisely, by setting each transition matrix on a pendant edge equal to 
the identity matrix: 

l<i j<fe 

We can write P = Mi (g) M2 <E) M3 (g) M4 • P, and observe that Qi2\3i acts as 

crP = Af^(i) ® M<,(2) ® M<,(3) ® M<,(4) ■ aP. 

Because permuting the transition matrices on the pendant edges will not change which quartet 
the tensor corresponds to, we need only consider crP, and we need only check the lemma for the 
elements (1324) and (13) (24), as these form a generating set for Qi2\zi- Referring to ([3]) and ([2]) 
we find that 

(1324) • PiiijiaU ~ ^i4,iz^ili2'^\iii'^i4,^ 

and 

(13)(24) •pH»2i3U = '^i3U<5M«2"^!iil'«3- 

Thus (1324)P = (13)(24)P, and we see that this tensor belongs to V'^^ (although it corresponds 
to a quartet rooted at the parent vertex of leaves 3 and 4). 

The lemma follows from a similar consideration for phylogenetic tensors belonging to V'^'^ and 
V^^ □ 

We note that there is an obvious analogous structure for the action of 5i3|24 and t/i4|23- 
Lemma |2 . II further illuminates our decision to study isotropy subgroups rather than automor- 
phism groups, and we believe that this reflects the underlying biology of the situation as well. For 
instance, it is clear that a phylogenetic method for quartets that returns the quartet tree 12|34 for 
a given data set should continue to return 12|34 even as the input sequences are permuted using 
elements of ^12134, whereas it is not possible to define an action of Aut(Ti) on the input sequences. 

3 Finding tree-informative invariants 

The space of homogeneous degree d polynomials VdiV®"^) carries a representation of 6„i defined 
by 

with ijj £ yS"". As an example, taking m = 4, (i= 2, we can write 

I«2«3«4jlj2j3j4 rlll2«3«4 J2J3J4J 

il ,...,44 Jl,--- J4 

(123)^1 o/(^)= 

1«2«3«4J1J2J3J4 lll2«4 2J41 

ii,...,i4,jl,...,j4 
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(12) 


(34) 


(12)(34) 


(13)(24) 


(14)(23) 


(1324) 


(1423) 


(12) 


e 


(12)(34) 


(34) 


(1324) 


(1423) 


(13)(24) 


(14)(23) 


(34) 


(12)(34) 


e 


(12) 


(1423) 


(1324) 


(14)(23) 


(13)(24) 


(12)(34) 


(34) 


(12) 


e 


(14)(23) 


(13)(24) 


(1423) 


(1324) 


(13)(24) 


(1423) 


(1324) 


(14)(23) 


e 


(12)(34) 


(34) 


(12) 


(14)(23) 


(1324) 


(1423) 


(13)(24) 


(12)(34) 




(12) 


(34) 


(1324) 


(14)(23) 


(13)(24) 


(1423) 


(12) 


(34) 


(12)(34) 


e 


(1423) 


(13)(24) 


(14)(23) 


(1324) 


(34) 


(12) 


e 


(12)(34) 



Table 1: The group multiplication table of G12 



34- 



so it is apparent that 



From Sumner et al\ 
tensors: 



[(123) ^ O f]i^i^i^i^j^j^j.^j^ — fi2i.3ilid2j33lU- 

we know that there exist degree d= 5 Markov invariants for quartet 



F:={/GP5(^^®')|g-'°/ = det(5)/} 



where g = Mi ® M2 (8) M3 (g) M4 and each Mi is an element of the Markov semigroup. Additionally, 
by considering the inner multiplication of Schur functions it was shown that dim(_F) — 4. Our 
purpose in the present work is to show how to find linear combinations of these invariants that 
are tree informative for a given quartet. 

Lemma 3.1. At a given degree d, the subset W C VdiV®"^) of phylogenetic invariants for a tree 
T is an invariant subspace under the action of Qq- . 

Proof Taking z e W, P e , a,nd a e Gt we have 



oz{P) = z{aP) = 0, 



because aP G by definition. 



□ 



Fo r example, it is clear by inspection that the quartet invariants given at the end of lEvans &: Speed 
(|l993l ) form an invariant subspace of 5i2|34, as required. At the end of this section we will examine 



the invariants given in that work more closely. 

In the context of this article we are interested in finding the subspace of Markov invariants that 
are simultaneously phylogenetic invariants for 7i, i.e. f d F such that f{P) ~ for all P G V'-^^ . 
As any invariant subspace must occur as a direct sum of irreducible modules, our immediate task 
is to identify the irreducible representations of tJi2|34- For convenience, in Table[l]we present the 
multiplication table of 5i2|34- 

Recall that the irreducible representations of a finite group can be put in one-to-one correspon- 
dence with its conjugacy classes, and the sum of the d i mens ion of each irreducible representation 
squared is equal to the order of the group (see ISaganI (|2001r ) for example) . Referring to Table [H 
we go ahead and explicitly compute by hand the conjugacy classes of 5i2|34- Wc find that there 
are five classes: 



[e] 
[(12)] 
[(12)(34)] 
[(13)(24)] 
[(1324)] 



= {e}, 

= {(12), (34)}, 

= {(12)(34)}, 

= {(13)(24),(14)(23)}. 

= {(1324), (1423)}, 



and thus conclude that there are five irreducible representations of ^?i2|34- It is satisfying to not e 
that this result can be confirmed using the combinatorial formula given in lOrellana et al. ( 20041 ) . 



6 





id sgn di 6,2 C 


[(12)] 
[(12)(34)] 
[(13)(24)] 
[(1324)] 


11112 
1-1-110 
1 1 11-2 
1 1-1-10 
1-1 1-10 



Table 2: The character table of ^/i2|34- 





id 


sgn 


(31) 


(2^) 


(21^) 




1 


1 


3 


2 


3 


[(12)] 


1 


-1 


1 





-1 


[(123)] 


1 


1 





-1 





[(12)(34)] 


1 


1 


-1 


2 


-1 


[(1234)] 


1 


-1 


-1 





1 



Table 3: The character table of &i. 



Additionally, we can infer that four of these representations are one-dimensional while the 
other is two-dimensional, as 1^ -I- 1^ -I- 1^ -I- 1^ -I- 2^ = 8 is the only 5 part partition of 8 into a 
sum of squares. We denote the four one-dimensional representations as id, sgn, di, d2, and the 
two-dimensional representation as C . 

It is useful to note that (12) (34) forms its own conjugacy class. This should be compared to 
the case for 64 where (12)(34), (13)(24) and (14)(23) form a single conjugacy class and is due to 
the fact that (12)(34) is a rotation, while ( 13) (24) and ( 14)(23) are reflections. Using the well 
known orthogonality relations for characters ( Saganl . [20011) . the character table of ^12134 is easy to 



derive and is presented in Table [21 

The reader is reminded that the conjugacy classes (and hence irreducible representations) of 
&rn are labelled by partitions of m with id = (1™) and sgn = (m). For convenience, we reproduce 
the character table of 64 in Table [31 

Recall that a (group) branching rule describe s the decorn position of the irreducible represen- 
tations of a group when restricted to a subgroup (IWevill95nf ). By staring at the character tables 



(Table [21 and Table [3]) and concentrating on the conjugacy class [(12) (34)] in S4 compared to the 
same class in ^12134, it is straightforward to derive the group branching rules: 

id id 
sgn sgn 

644^12134: {31}^C + d2 (4) 
{2^} ^ id + sgn 
{212} _^c + di. 

Given that is a module for 64 J, ^i2|34, we would like to examine the structure of Markov 
invariants in each irreducible module thereof. This will reveal exactly when an invariant is tree- 
informative. 



Recall that the primitive idempotents ([Procesi 120071 ) of the group algebra C [Q] are 



where x is an irreducible character. These primitive idempotents satisfy the orthogonality con- 
ditions ■ Q^> = S^^'Q^, and, given a ^/-module V, project onto the irreducible subspaces of 
V. 
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*7 ^ 012 34 


rrP^ 






6 


^1 


^2 


Pn 






pr 
^3 


pr 


(34) 


Pi 


h 




(12)(34) 


Pi 


pr 
^2 


pr 
^3 


(13)(24) 


pr 


h 


pr 
^3 


(14)(23) 


pr 


pr 
^2 


h 


(1324) 


pr 


pr 
^3 




(1423) 


pr 


h 


^2 



Table 4: Action of Gi2\3i on trimmed tensors. 



We are, of course, interested in ^ = Qi2\3a a-nd will consider properties of an arbitrary f (z F 
under the projections o / for each irreducible character of 0i2|34- In what follows we use the 



fact that x(ct ^) — xi'^) for finite groups, thus 



x{(T)a 1 o /, 



where the second equality holds because the map a ^ a ^ is simply a permutation of the group 
elements and the third equality holds because the irreducible characters of t/i2|34 are real. 

For convenience we take the trimmed tensor Pi G V'-^^ as before with root placed at the parent 
vertex of leaves 1 and 2. This tensor has components 

Define the "reflected" trimmed tensor as 

PI = (13)(24)A, 

so that Pf is obtained by moving the root vertex to the parent of leaves 3 and 4. The trimmed 
tensors P2, P3 and their reflected counterparts PJ, PJ are defined similarly. In Tabled] we explicitly 
record the action of Qi2\3a on each of these trimmed tensors. 

Now using the character table for Qi2\3i, we can infer any tree-informative identities that occur 
between the values of 8^ o /(P^) for i = 1, 2, 3 and each irreducible character x- 

For the id representation we have 



Bid o /(Pi) := 



- E 



Xid(a)/(aPi 



''■G<?12|34 



i [/(Pi) + /(Pi) + /(Pi) + /(Pi) + /(P[) + /(P[) + /(PO + /(P[) 
/(Pi) + /(Pn 



eMo/(P2):=i E Xid(a)/(aP2) 



o-e5i2 



\ [f{P2) + /(PJ) + /(P3) + f{P^2) + f{P2) + f{P2) + f{P^) + f{P3) 
fih) + f{PS) + fih) + f{PD 
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and 



"■£012134 

= I [fih) + f{p;) + fih) + f{PD + fiPD + fih) + fih) + an) 

= 7 \fiP3) + /(^2 ) + fih) + an) 



We see that this representation is not tree-informative. 
For the sgn representation we have 



esgnO/(Pi) 



0'GSi2|34 

\ [f{Pl) - /(Pi) - /(Pi) + /(Pi) + /(P[) + f{Pl) - fiPl) - fiPl. 



Osgn O /(P2) := ^ E Xsgn(^)/(^P2) 

0-G5i2|34 

= I [fih) - f{PD - fih) + fin) + f{P2) + f{p;) nn) - nh) 



and 



©sgn ° fiPs) 



I E Xsgn(^)/(^/'3) 

0'eSl2|34 

^ [/(^3) - f{PD - /(^2) + /(PJ) + /(PJ) + /(P3) - /(P2) " f{P^) 

fih) + f{PD - fih) - f{PD 



Thus in this case we have 9sgn o /(Pi) = and 9sgn o /(P2) = — 6sgn o /(-Fa), so that this 



representation is tree-informa tive. A major outco me of this article is that these are exactly the 
relations that were derived in Sumner et al. I (|2008l) by explicit computation. 
For the di representation we have 

©rfi°/(A):-^ E xMfiTPi), 

0'eSl2|34 

= I [.f(Pi) /(Pi) - /(Pi) + /(Pi) - /(P[) - /(PH + /(P[) + .f(Pl) 
= 0, 



erf,o/(P2):=- E XdA'^)f{'jP2), 

0'GSi2|34 

= ^ [/(/^2) - /(PJ) - /(P3) + fiPD - fih) fiPD + fiPD + fih) 
= 
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and 



"■£612134 

= I [/(-P3) - m) - m) + f{ps) - m) - m) + m) + hpd 



=0. 



We see that this representation vanishes on every quartet. 
For the d2 representation we have 



"■£012134 

= \ [f{Pl) + f{Pl) + f{Pl) + I{Pl) f{P{) f{Pl) - f{P[) f{Pl)_ 



"£512134 



and 

0d2°/(P3):=^ E X<i2W/(^P3), 

"■65l2|34 

= ^ [/(-Ps) + /(-P2 ) + /(P2) + f{Pi) - f{Ps) - fih) - fih) - m) 
= 0. 

This representation vanishes identically on the quartets 13|24 and 14|23 but not on 12|34. 

As the C representation is 2-diniensional we consider a tuple / := (/i,/2) 1— > ©c ° / with 

©co/(Pi):=^ Y Xc(<7)/(<7Pi) = i[2/(Pi)-2/(Pi)] =0, 

"■6512134 

©co/(P2):=i Y Xc(a)/(aP2) = ^[2/(P2)-2/(PJ)]=i[/(P2)-/(P2^ 

"6012134 

and 

Qc o /(P3) ■■=1 Yl Xc{cr)f{aPs) = i [2/(P3) - 2/(P3-)] = 1 [/(P3) - /(P[)' 

"£012|34 

In this case the representation vanishes identically on the quartet 12|34 but not on the other two 
quartets, and is hence tree-informative. 

It is worth noting that the above relations are generic statements about invariants that belong 
to particular irreducible modules of Gi2\?a ^nd it is still possible for there to be additional tree- 
informative relations. For example, in the id case it is clear that an invariant could be tree- 
informative if it so happened that /(Pi) + f{P[) = 0. 
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It seems that the tree-informative Markov invariants identified in Sumner et al. I (Hooi) trans- 
form under the sgn representation of ^i2|34- Unfortunately, our understanding of the Schur-Weyl 
duahty does not allow us to take the final step and directly write as a sum of irreducible mod- 
ules of 64. This is because the d etails of the 6 4 symm etry seems to get lost in the derivation of 



the existence conditions given in 'Sumne r et 'd\ l|2008l) . However, in the next section will give a 
procedure that generates invariants in F that have clear transformation properties under 64. As 
it will be clear which modules these invariants belong to, we need only give a linearly independent 
set of four invariants to infer the decomposition of F into irreducible modules of S4, and whence 
of 012 1 34 using the group branching rules 

Before we do this however, we return to the invariants o fH vans fc Speed! (|l993l) and explicitly 



show how they occur as irreducible modules of Qi2\'M- As an illustration of the power of the present 
approach, we can even do this without delving into the precise meaning of the formal expressions 
they give for th eir invariants. 



In Section 7 lEvans fc Speed! (jl993f ) give phylogenetic invariants for the Kimura 3ST model on 
the quartet tree 12|34 in three forms 

z^'^^X, X') ■■= E [(Fi +Y2,x) {Y^ + Yi, x')] - E [{Yi + i^., x)] E m + x')] , 
z^^ (X, x') E ^ ^ (j2 ^'^ 

- E m +Y2,X) {Y3 + x'>] E m + Y2,x') {Y3 + n, x)] , 
X) E [{Y, + Fa, x) {Y2 + Y^, x')] E [{Y, + Y2,x') {Y3 + n, x)] 

- E [(Fi + n, x) {Y2 + Y3, x')] E m + n, x'> {Y2 + y3,x)] , 

where E denotes expectation, Yi,Y2,Y3, Y4 are the random variables at the leaves of the quartet 
taking on values in the abelian group Z2 x Z2 and x,x' ^re non-trivial characters of Z2 x Z2. 

Now it is clear by inspection that z^''^ transforms as the id representation of t?i2|34 and z^'^'^ 
transforms as the sgn representation. For z*^°) we observe that z^°''>{x,x) transforms as the id 
representation, as does the symmetric combination z'^''-'(x, x') + z^°'Hx' 1 x)- Finally, by inspecting 
Table [5] we see that the anti-symmetric combination ^^''•'(x, x') ~ ^^"'Hx'iX) transforms as the 
di representation. In this way we have completely characterized these quartet invariants into 
irreducible modules of Gi2\3'i- 



4 Explicit forms 

In this section we present a trick that freely generates Markov invariants, and we apply the previous 
theory to identify which C/12134-module these invariants belong to. We conclude by identifying F 
as a sum of irreducible (yi2|34-modules. 

We begin by observing that the (completely antisymmetric) Levi-Citiva tensor 



Eiiiaiau Sgn(ii ^21314) 

transforms as the sgn representation of the general linear group GL(C^) 
GL(C4), 

I<il.j2j3 J4<4 



That is, for any g G 



Now, in a procedure that is consistent with that given in ISumner et al\ (|2008[ ) (we only ignore 
symmetrization across the rows of associated tableaux), we can freely construct Markov invariants 
such as 

/(t/)) = ^ ^ 'll^T.T,i3ii'4^jij2T,T.'4^kik2k3ki'4^lil2l3li'^rnim2m37iii^jikilimi^j2k2l2 -i 
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where each subscript "S" can be thought as either a su m over states ( as wit h Allman fc Rhode"i 



or the "0" component of the basis specified in I Sumner et al. and all remaining 



indices are summed from 1 to k. One can readily check that if 

V'iii2i3U ^ V'iiiaisU ^ 5Z '^Lji ™L]2 "^isjs "^L]^ '^51525354 J 

l<jl J2 ,j3,i4<fc 

with each to,-"^ a Markov matrix such that Y^- m,-"^ — 1, that 

fi^jj') = det(Afi) det(M2) det(M3) det(M4)/(V'), 

as required. Note that this construction requires that the S's are spread evenly across the legs of 
the tensors (one for each part of the tensor product). 

It is worth observing that this presentation can be related to that given bv I Allman fc Rhodes! 
by observing that the cofactor matrix can be expressed as 



C0f(M)]^jj — 'mi-^i2mjj^j2mkik2^iijikiaii2j2k2b- 



However, in that work the phylogenetic invariants constructed were not required to have any 
particular transformation properties under the action of the Markov semigrou p. It would also be 
of inte rest to determine the transformation properties of the invariants given in Allman fc Rhodei 



(|2003l ) under the relevant isotropy subgroup. 

In the general case of k states, the Levi-Citiva tensor has k legs, thus the minimum degree we 
can construct an invariant as above is d = k. However, by anti-symmetry this only works for even 
m, and we can const ruct a single d — k Markov invariant for each even m. This is consistent with 
Sumner et all ( 2008! ) where it was observed that there exist Markov invariants of degree d = k for 



even m only. For m = 2 the corresponding Markov invariant forms the foundation of the Log-Det 
distance estimator, and m — A the Markov invariant is referred to as the "quangle". 

Taking the quartet case to = 4 and c? = A: + 1, we must insert a total of four E's into the 
expression for the Markov invariant (one for each leg of the tensor product). If we represent the 
five factors in the expression as boxes /, J, K, L and M, we are asking how many ways are there 
to put four objects {1,2,3,4} into 5 identical boxes. Clearly, for each set partition of {1,2,3,4} 
this can be done in the various ways given in Table [S) For example, we have 

y(12,34)^^-j = ^ V'SSi3i4V''jij2SsV'fcifc2fc3fc4V'iii2i3i4V'mim2m3m4 

EjifciZirrii £j2fc2'2m2 ^^3^3131713 ^iikilinni ) 

and 

j(12,3,4)^^-j = ^T,T.i3iA'4']i]2^jii'kik2k3T.'^nil2l3Ui'mim2m3mi 

■ ^jikilimi £j2fc2'2m2 ^i3fc3'3™3^i4i4'4m4- 

Now given that the rows in each set partition can be interchanged freely, it is easy to check 
that under 64 these invariants transform amongst each other following the permutations, e.g. 
a ■ (ijk,l) — {a{i)a{j)a{k),ull)). In fact one can explicitly check that 

(124) o /(12,34) ^ j(24,13) ^ y(13,24)_ 

Thus for each set partition, the corresponding invariants form an invariant subspace of 64. As 
is depicted in Table [51 we label these invariant subspaces by enclosing the partition shape within 
square brackets [•] . That is, 

where (•,...,•) denotes linear span. 
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It is too much to hope that for each set partition that the corresponding invariant subspace 
will be irreducible, but using the primitive idempotents of 64 it is a straightforward pencil and 
paper computation to show that for the [4] module we have 

0(31) ° f^''''^ = 0(2.) o ^ 0^^^,^ „ ;(1234) ^ 0^^^ „ ^(1234) ^ q 

For the [31] module we note the 64 symmetry so we need only consider the canonical example 

Oid o f^^'^^'^^ = i (/(123.4) J(124,3) ^ y(134,2) ^ j(234a)-) ^ 
0(31) O /(123,4) ^ !_ (3^(123,4) _ ^(124,3) _ ^(134,2) _ ^(234.1)) ^ 

0(2.) o = 6(21.) o /(i^^'^) = e.,„ O /(123,4) = 0, 

with obvious similar relations for f''-^^^'^\ y(i34,2) g^^^j j(2344)_ the [2^] module we can again 
exploit the 64 symmetry and consider 

Oid o = 1 (J(12,34) _^ j(13.24) ^ ^^(14,23)) ^ 

0(2.) O /(12-34) = i (2/(12,34) „ ^(13,24) _ ^^(14,23)) 
0(31) o /(12.34) ^ 0^^^^^ „ ^(12,34) ^ 0^^^ „ ^(12,34) ^ 

with obvious similar relations for /(i3,24) ^^^^ j(i4,23) Similarly, for the [21^] module we have 

Old o /(l^'^^'') = i (/(12,3,4) _,_ j(13,2,4) _,_ j(14,2,3) _^ j(23,l,4) _^ j(24,l,3) _^ y(34,l,2)^ ^ 
0(31)°/™-H/^''''''^-/'''''^'^)' 

0(2.) O /(12'^'^) = (2(/(12,3.4) ^ j(34a,2)^ _ (y(13,2,4) _^ y(24,l,3) _^ y(14,2,3) ^_ j(23,l,4)^-) ^ 

Ci „ /•(12.3,4) o „ /•(12.3,4) n 

0(21.)0/^ '^^BsgnO/^ '^=0. 

Finally, for the [l4] module: 

0idO/(1^2,3,4) = J(l,2,3,4)^ 

C ^ /•(1, 2,3,4) o /•(1,2,3,4) o ^ /•(1, 2,3,4) o ^ ^(1,2,3,4) p. 

0(31) O ■ • ■ ' = fc)(2.)/' ' ' ■ ' = C'(212) O ■ ' ' = Bsgn O ' • ^ = U. 

Thus as irreducible modules of 64, we have 

[4] - id, 
[31] = id ©(31), 
[2^] ^ id ©(22), 
[21^] ^ id©(22)©(31), 
[1^] = id. 

It is also worth noting that the dimensions of these modules add up the the number of invariants 
given in Table [5] 

However, we know that F is only 4 dimensional, so we have far too many invariants. To help 
rectify this, we note that 

/*■ HV') = ^ ] V'SSSsV'jl j2 33 34 V'fclfc2fc3fe4''/'hi.^3i4''/'"ll™2"l3™4^3lfcl^l™l ■ ' ■ £j4fe4'4ni4; 

which can be factorised into a degree d — A invariant multiplied by the "trivial" invariant $(V') ■— 
V^ssss- Thus, [4] e $ • 7?4(y»4)x*GL(y)^ conclude that 

F = [4] © F, 
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Table 5: Classes of invariants: [4], [31], [2^] , [21^] and [l^] . 



with dim(/^) — 3. 

A t this point we throw our hands in the air and resort to explicit computation with R (|R Development Core Teainl . 
20061 ) (code available upon request) to show that 



[4] ^[31],,, 

[2 ] (22) — ] (22) 1 

[31](31) ^ 0, 

[211(31)^0' 

where [-j^.^ denotes the (•) 64-module contained in [■]. From this we can conclude that 

^=[4]©[l1®[2l(22)- 
So that, as a decomposition into irreducible representations of ©4, we have 

i^ = 2-ide(22). 

Referring to the branching rule 64 J, ^i2|34, as a decomposition into irreducible modules of 
^i2|34 we see that 

= 3 ■ id © sgn. 

Thus we have achieved our main aim of expressing _F as a direct sum of irreducible modules of 64 
and 0i2|34- 

By decomposing F into a direct sum of irreducible modules of ^i2|34 we have shown that there 
is a single copy of the sgn representation and hence a single tree-informative Markov invariant for 
the quartet 7i := 12|34. Using the primitive idempotent of the (2^) representation of 64 we have 

©(22) O /*'^''^' = ^ (2/(13.24) _ J(14,23) _ ^(12,34)^ 

Now projecting further with the sgn representation of 5i2|34 we get 

©.gn O i (2/(13,24) _ J(13.23) _ ^(12,34)) ^ 1 (^(13,24) _ ^(14,23)) 

and we define 

Q, l(/(13,24)__^(14,23)). 
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We can use the action (14) ■ 7i 72 to transform this invariant to produce a tree- informative 
invariant for 

02 := 1(/(14,23)_J(12,34))^ 

and similarly to produce a tree informative invariant for T^: 

03 :=i(/(12.34)_j(13,24)). 

These are none other t han t he Markov invariants referred to as the "squangles" (stochastic quan- 
gles) in lSumner et al\ (|2008l ). 

Similar considerations reveal that the three Markov invariants that transform as the id rep- 
resentation of f/i2|34 are /'■^^^''^ j(i>2,3,4)^ j(i2,34)^ summarize all of this in the following 
theorem. 

Theorem 4.1. The set of Markov invariants for quartet trees 

F := {/ e V,{V®^) I g-' o / = dct(5)/} , 

where g = AIi M2 (Ei M3 (g) AI4 and each Mi is an element of the Markov semigroup, can be 
decomposed into irreducible modules of 64 as 

F = 2- id® (2^) 

= (/'^^^''^) © ^j(1.2.3,4)\^ ^ ^2/(12,34) _ j(13,24) _ y.(14.23) ^ 2/(^3,24) _ j(12,34) _ j(14,23)^ ^ 

and irreducible modules o/C7i2|34 as 
F = 3 ■ id® sgn 

= ^ y(1234)\^ ^ ^ j(l,2,3,4)\^ ^ ^ j(12,34)\^ ^ ^ j(13,24) _ j(14,23)\^ ^ 



As a final loose end, we note that a crucial aspect to the performance of the Markov invariants in 
the simulation study given in lSumner et ^ (l2008l) was the observation that 



01(^^2) > 0, 

with similar relations for the other invariants. Now we have explicit forms for the invariants we 
can easily derive the relevant relations. Consider, consistent with 72, the "trimmed" phylogenetic 
tensor P with components Pi^^i^iau = Si^i3Si2iifpiii2 where T^iiis = '^n'^i^- Now 

ji ' {P) — P'Si2'SiiPji'Sj3YlPkik2k3kiPlil2l3hPmim2rn3m4^jikilirni^i2k2l2rn2^j3k3l3rn3('i4kil4,m4 
= ^■Si2^i2u'^jl^^Jlj3^kik2'>kik3Sk2kii^hl2^lil3^l2h^mim2^mim3Sm2m4. 

= V'Ei2'0JlsV-'feifc2V'ii;2''/'mim2kjifciiimi ||ei2fc2i2m2L 



and similarly 



j(14,23)j^p^ — i^\i2'^kik2^lll2'^mim2\<^jlkilimi\\e-i2k2l2m2 



It is clear that '0jii2 — ^^i2'^ji^ for all ji,i2, and we have the required result. 

With our newly computed forms of the squangles expres sed using the Levi-Citiva tensor, we 
repeated the simulation study given in Sumner et al. ( 20081 ) and yielded identical results. This 
gives a strong experimental confirmation of the theory underlying this work, as the previous form s 
of the squangles were computed using the Young tableaux procedure given in lSumner et al. ( 2008f ). 
Also wc note that the tree- informative squangles are actually linearly dependent: 

Oi + O2 + O3 = 0. 



This refines the results given in ISumner et all (|2008l ) where this dependence was not observed. 
This was missed because of the obscure nature of the basis used in the construction of the Young 
tableaux. Hopefully this article has helped to illuminate some of these issues significantly. 
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5 Discussion 



In this article we have applied the representation theory of the isotropy subgroup of leaf permu- 
tations on a quartet to give a systematic procedure for finding tree- informative invariants. In the 
quartet case we applied this to Markov invariants and reproduced from theoretical considerations 
relations that were previously derived computationally. 

For general unrooted binary trees the corresponding isotropy groups arise as combinations of 
direct and "wreath" products of 62 and 63. For example, in the quartet case ^i2|34 — 62 I &2, 
and for the (balanced) binary tree with 6 leaves and 3 cherries we have ^i2|34|56 — ©2 I S3. It 
would be fruitful to continue to study the representation theory of wreath product groups with an 
eye applications to phylogenetic problems. In particular, it is worth noting here that the isotropy 
subgroup for "caterpillar" (completely unbalanced) trees is isomorphic to the quartet case. Thus 
the theory we have developed in this article will apply directly in that case with complication 
in detail only, as there are more invariants and more trees to check against for linear relations. 
Additionally, for the case of comple t ely ba lanced rooted trees, the irreducible representations have 



been enumerated in lOrellana et al\ (|2004l ) 



Using leaf permutations we have been able to explicitly incorporate the underlying tree struc- 
ture into the analysis of tensor-based approaches to phylogenetic problems. This is surely a step 
fo rward, but the r e rem ains a gap between the work presented in this article and that presented 
in Sumner et all ( 2008 ). That is, one would like to derive the decomposition of the module of 



Markov invariants into irreducible modules of the tree isotropy groups directly without the need 
for any explicit computation. This was not quite achieved in this article and presents itself as an 
open problem. 

More generally, the opportunity exists to derive a general duality between representations of the 
Markov semigroup and those of tree isotropy groups. This would be in analogy to the Schur-Weyl 
duality between representations of the general linear and the symmetric group. 
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